3.2 Expected Failures

Some bugs are complicated to fix, or not very important, and are left as known bugs. If there is a test case that triggers the bug and fails, ERT will alert you of this failure every time you run all tests. For known bugs, this alert is a distraction. The way to suppress it is to add :expected-result :failed to the test definition:

(ert-deftest future-bug ()
  "Test `time-forward' with negative arguments.
Since this functionality isn't implemented, the test is known to fail."
  :expected-result :failed
  (time-forward -1))

ERT will still display a small f in the progress bar as a reminder that there is a known bug, and will count the test as failed, but it will be quiet about it otherwise.

An alternative to marking the test as a known failure this way is to delete the test. This is a good idea if there is no intent to fix it, i.e., if the behavior that was formerly considered a bug has become an accepted feature.

In general, however, it can be useful to keep tests that are known to fail. If someone wants to fix the bug, they will have a very good starting point: an automated test case that reproduces the bug. This makes it much easier to fix the bug, demonstrate that it is fixed, and prevent future regressions.

ERT displays the same kind of alerts for tests that pass unexpectedly as it displays for unexpected failures. This way, if you make code changes that happen to fix a bug that you weren’t aware of, you will know to remove the :expected-result clause of that test and close the corresponding bug report, if any.

Since :expected-result evaluates its argument when the test is loaded, tests can be marked as known failures only on certain Emacs versions, specific architectures, etc.:

(ert-deftest foo ()
  "A test that is expected to fail on Emacs 23 but succeed elsewhere."
  :expected-result (if (string-match "GNU Emacs 23[.]" (emacs-version))
                       :failed
                     :passed)
  ...)